Sound and Efficient Dynamic Verification of MPI Programs with Probe Non-determinism

We consider the problem of verifying MPI programs that use MPI_Probe and MPI_Iprobe. Conventional testing tools, known to be inadequate in general, are even more so for testing MPI programs containing MPI probes. A few reasons are: (i) use of the MPI_ANY_SOURCE argument can make MPI probes non-deterministic, allowing them to match multiple senders, (ii) an MPI_Recv that follows an MPI probe need not match the MPI_Send that was successfully probed, and (iii) simply re-running the MPI program, even with schedule perturbations, is insufficient to bring out all behaviors of an MPI program using probes. We develop several key insights that help develop an elegant solution: prioritizing MPI processes during dynamic verification, handling nondeterminism, and safe handling of probe loops. These solutions are incorporated into a new version of our dynamic verification tool ISP. ISP is now able to efficiently and soundly verify larger MPI examples, including MPI-BLAST and ADLB.